MSFT-0579/167505.2 38 PATENT 

What is claimed is: 

1 . A method of classifying data according to perceptual properties of the data, the 
method being suited for searching and sorting large databases of media entities, including 
music, video and image databases, the method comprising: 

assigning to each media entity of a plurality of media entities in a data set to at least 
one class, each class of said at least one class corresponding to a subset of perceptual 
properties pre-defined for the data set; 

processing each media entity of said data set to extract at least one digital signal 
processing characteristic for each media entity; 

generating a plurality of feature vectors for said plurality of media entities, wherein 
each vector includes said at least one class and said at least one digital signal processing 
characteristic; and 

forming a classification chain based upon said plurality of feature vectors. 

2. A method according to claim 1, further comprising calculating a neighborhood 
distance within the vector space of said classification chain for each of said at least one 
perceptual class. 

3. A method according to claim 2, wherein said neighborhood distance defines a 
distance within which two vectors in the classification chain space are in the same 
neighborhood. 

4. A method according to claim 2, wherein said calculating of a neighborhood distance 
for each of said at least one perceptual class includes determining a distance within which two 
vectors of the classification chain possess the same class given a threshold degree of error. 

5. A method according to claim 4, wherein a human determines said threshold degree of 
error that will be tolerated, said threshold degree of error placing a maximum limit on the 
distance that may be used for determining neighborhoods. 



6. A method according to claim 1, further comprising: 
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processing an unclassified media entity to extract at least one digital signal processing 
characteristic for the unclassified media entity; 

generating a vector for the unclassified media entity including said at least one digital 
signal processing characteristic; 

presenting the vector for the unclassified media entity to the classification chain; and 

classifying the unclassified entry with an estimate of at least one perceptual class by 
calculating the representative at least one perceptual class of the subset of the plurality of 
vectors of the classification chain located in the neighborhood of the vector for the 
unclassified entity. 

7. A method according to claim 6, wherein said classifying of the unclassified entry 
includes locating the subset of the plurality of vectors of the classification chain located in the 
neighborhood. 

8. A method according to claim 7, wherein said locating includes calculating, as the set 
of vectors of the plurality of vectors of the classification chain within the neighborhood 
distance, the set of vectors of the plurality of vectors of the classification chain within a 
maximum Euclidian distance for classifying new unclassified entities within a threshold 
tolerance. 

9. A method according to claim 8, wherein a human determines said maximum 
Euclidian distance for purposes of determining the neighborhood vectors within the threshold 
tolerance. 

10. A method according to claim 6, wherein said classifying of the unclassified entry 
includes classifying the unclassified entry with a median class represented by the 
neighborhood. 

11. A method according to claim 6, wherein said class is described by a numerical value 
and said classifying of the unclassified entry includes classifying the unclassified entry with a 
mean of numerical values found in the neighborhood. 
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12. A method according to claim 6, wherein said classifying includes returning at least 
one number indicating the level of confidence of the class assignment. 

13. A method according to claim 12, wherein said classifying includes returning a ratio 
defined by the number of entries in the neighborhood having the representative class over the 
number of entries in the neighborhood not having the representative class. 

14. A method according to claim 12, wherein said class is described by a numerical value 
and said classifying includes returning a standard deviation of the numerical values in the 
neighborhood. 

15. A computer readable medium bearing computer executable instructions for carrying 
out the method of claim 1 . 

1 6. A modulated data signal carrying computer executable instructions for performing the 
method of claim 1. 

17. A computing device comprising means for performing the method of claim 1 . 

1 8. A computing system, comprising: 
a computing device including: 

a classification chain data structure stored thereon having a plurality of classification 
vectors, wherein each vector includes data representative of at least one perceptual class as 
classified by humans and digital signal processing data as classified by at least one computing 
device; and 

processing means for comparing an unclassified media entity to the classification 
chain data structure to determine at least one perceptual class of said unclassified media 
entity. 

19. A computing system according to claim 1 8, wherein said comparing includes 
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determining a set of classification vectors that are within a neighborhood distance from a 
vector represented by said unclassified media entity. 



20. A computing system according to claim 19, wherein said neighborhood distance 
defines a distance within which two vectors in the classification chain space are in the same 
neighborhood. 

21 . A computing system according to claim 20, wherein said neighborhood distance is 
determined for each of said at least one perceptual class, and said determination includes 
determining a distance within which two vectors of the classification chain data structure 
possess the same class given a threshold degree of error. 

22. A computing device according to claim 21, wherein a human determines said 
threshold degree of error that will be tolerated, said threshold degree of error placing a 
maximum limit on the distance that may be used for determining neighborhoods. 

23. A computing system according to claim 19, wherein said processing means classifies 
the unclassified entry with an estimate of at least one perceptual class based upon the vectors 
within the neighborhood of a vector representing the unclassified entity. 

24. A computing system according to claim 23, wherein said classifying includes 
classifying the unclassified entry with a median class represented by the neighborhood. 

25. A computing system according to claim 23, wherein said at least one perceptual class 
is described by a numerical value and said classifying of the unclassified entry includes 
classifying the unclassified entry with a mean of numerical values found in the neighborhood. 

26. A computing system according to claim 23, wherein said classifying includes 
returning at least one number indicating the level of confidence of the class assignment. 



27. 



A computing system according to claim 26, wherein said classifying indudes 
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returning a ratio defined by the number of entries in the neighborhood having the 
representative class over the number of entries in the neighborhood not having the 
representative class. 

28. A computing system according to claim 26, wherein said class is described by a 
numerical value and said classifying includes returning a standard deviation of the numerical 
values in the neighborhood. 

29. A computing system according to claim 1 8, further comprising: 
an input device for receiving a new unclassified media entity; and 

an output device for outputting at least one perceptual class of said new unclassified 
media entity based upon said classification chain based upon processing of said processing 
means. 

30. A classification chain data structure utilized in connection with the classification of 
new unclassified media entities, comprising: 

a plurality of classification vectors, wherein each vector includes: 
perceptual data classified by humans; and 

digital signal processing data classified by at least one computing device. 

31. A method of generating a classification chain having a plurality of vectors describing 
a plurality of media entities, comprising: 

assigning, by an expert, a first value to a media entity according to a pre-defined 
perceptual characteristic of media entities; 

assigning, by a computing system, a second value to the media entity according to a 
pre-defined digital signal processing characteristic; 

generating a vector based on at least said first value and said second value; and 

adding said vector to a classification chain data structure. 

32. A method according to claim 3 1 , wherein said method is repeated until a sufficient 
number of vectors have been added to said classification chain data structure, such that said 
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classification chain data structure successfully classifies unclassified media entities within a 
threshold degree of success. 

33. A method according to claim 32, further comprising: 

inputting a vector representative of a new unclassified media entity; 

comparing said vector to the vector space of the classification chain data structure; 

and 

outputting an estimate of the perceptual class of the new unclassified media entity 
based upon vectors found in a neighborhood of the classification chain data structure, wherein 
a neighborhood of vectors is defined as a set of vectors located within a neighborhood 
distance. 

34. A method according to claim 33, further comprising: 

outputting an estimate of the confidence level with which said estimate of the 
perceptual class is correct. 

35. A method according to claim 34, wherein if said estimate of the confidence level is 
low, a human examines the new unclassified media entity and said human determines an 
action for said classification chain data structure based thereon. 

36. A method according to claim 35, wherein said human adds a new subset of perceptual 
properties to the classification chain data structure defined by the new unclassified data 
structure. 

37. A method according to claim 35, wherein said human modifies an existing subset of 
perceptual properties represented by the classification chain data structure in accordance with 
the results of the human examination. 



38. A method according to claim 35, wherein the new unclassified media entity is rejected 
from the classification chain as an outlier. 
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39. A method according to claim 35, wherein the performance level of the classification 
chain improves over time due to the examination of unclassified media entities that have a 
low confidence level associated therewith. 



40. A method according to claim 33, wherein it is determined whether said vector 
representing said new unclassified media entity has been previously processed, thereby 
obviating the need to re-process said vector via the classification chain. 



